Repository navigation
fix(client-runtime): returning to the app no longer restarts the reconnect backoff for an unreachable host - #24064
Batuhan1999 wants to merge 1 commit into
Conversation
…nnect backoff for an unreachable host Every application-active wakeup ended the retry wait and reset the retry ladder to its first rung. On desktop that fires on each visibilitychange to visible, so a saved SSH environment whose host no longer resolved retried every few seconds for as long as the app was in use and almost never reached the 5 minute cap. A wakeup still ends the wait and retries at once, but an application-active wakeup (desktop or mobile) now resets the ladder at most once per RETRY_MAX_DELAY_MS. A network change still resets it every time. Refs pingdotgg#15616 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ApprovabilityVerdict: Approved at Macroscope's review found this PR approvable — This focused client-runtime fix prevents repeated foreground wakeups from restarting the reconnect backoff while preserving immediate retries and unconditional network-change resets. The added TestClock coverage verifies all wakeup variants and the five-minute reset behavior. You can add or adjust custom eligibility rules. Learn more. |
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info
📝 Walkthrough
Merge Risk: ⚪ Minimal · up to
|
|
Another real-world case for this, on iOS, where the cost is higher than extra retries. Setup: T3 Code iOS 2.0.0 (App Store), iOS 27.0, Tailscale iOS 1.104.4. Two environments: a Linux server that is always on, and a MacBook that is asleep most of the day. Both are reached over Tailscale. What happened: while the Mac slept, the app kept retrying it (the UI kept flashing "reconnecting to "). On iOS, that traffic to an unreachable tailnet peer triggers tailscale/tailscale#21773: wireguard-go's 64-buffer pool fills up and the whole tunnel stalls for ~100 s. That includes the healthy Linux server, so the app lost its working environment as well, every 2–3 minutes. Confirmed by the on/off switch from #11478: switching the Mac environment off stopped the stalls at once. There were none in the next 17 minutes of heavy use, against one every 2–3 minutes before. Over the following 9 days the Mac was offline in 165 hours, with the environment kept off, and the phone's tunnel restarted only twice. The switch works, but only once you know the cause. A first-time user just sees T3 dropping its good environment. So I'd like to see this PR merged. A ladder that actually reaches its 5-minute cap on a host that stays down would keep the traffic below what tailscale/tailscale#21773 needs: per that issue, packets to the dead peer have to keep arriving more often than every ~90 s. Related: #11236 (same report, closed after the on/off switch landed), tailscale/tailscale#21773. |
|
This PR conflicts with See the conflict window. |
|
The VPN toggle that recovers from tailscale/tailscale#21773 may also count as a |
|
I’m seeing this repeatedly on T3 Code iOS 2.3.0 with two macOS hosts, both on 0.1.11 and paired via Tailscale HTTPS ( This looks similar to #37699, but I haven’t yet checked whether Safari or another tailnet service also fails during an incident, or whether one host becoming unreachable triggers it. So I can’t tell whether this is the same tunnel-wide stall. Would an iOS Tailscale bug report ID and same-device Safari results captured during the next incident help distinguish that from an app-only reconnect issue? |
|
@mochafreddo Those symptoms match what I saw in #37699. The app said
A Tailscale bug report ID is most useful on tailscale/tailscale#21773, since only Tailscale can read those logs. The Safari and ping results are the useful part here, because they show whether keeping the backoff in this PR would be enough on its own. |
|
@mmmbuford Thanks for the detailed explanation and troubleshooting steps! I'll check Safari connectivity and run I'll also check my Tailscale iOS version and report back with the results. |
Adding another desktop data point for this fix. Setup: macOS desktop Observed during a 3.7-hour offline window:
This also lines up with @Batuhan1999 Since this PR currently has |
Problem
When a saved environment stays unreachable, the reconnect backoff from #14897 should climb to its 5-minute cap. In practice it rarely gets past a few seconds while the app is in use.
Real telemetry from one macOS desktop install (nightly
0.0.46-nightly.20261011). It had a saved SSH environment whose hostname no longer resolved. Between Oct 20 16:59 and Oct 23 11:33 local time:desktop.ipc.sshEnvironment.ensureEnvironmentfailed 2,564 times (de-duplicated). All but one failure wasssh: Could not resolve hostname <host>.Mechanism on
main:apps/web/src/connection/platform.tssends anapplication-activewakeup on everyvisibilitychangeto visible.resetsRetryBackoffinpackages/client-runtime/src/connection/wakeups.tsreturns true for every application-active variant and fornetwork-changed.EnvironmentSupervisor.run, a wakeup during backoff ends the wait, and thenresetRetryLadder()setsfailureCount = 0. So each time the app comes back to the foreground, the next delays start again at ~1–2 s.Change
This is a change to
packages/client-runtime/src/connection/supervisor.tsonly, so web, desktop, and mobile all pick it up.RETRY_MAX_DELAY_MS(5 minutes). The supervisor records when the last reset from a wakeup happened, usingClock. Within that window, a wakeup still causes an immediate attempt, but a failure afterwards continues up the ladder instead of starting again at the first rung.network-changedstill resets the ladder every time, because a network change can make the host reachable.application-active-probeandapplication-active-reconnectare handled the same way as desktopapplication-active. Each one means the user came back to the app, not that network conditions changed. A long mobile resume still replaces a live session, or interrupts an attempt in progress, and resets the ladder immediately (connectedLeaseEnd/waitForEstablishmentInterrupt). That path is not changed, because it handles sockets the OS killed in the background, not a host that stays down.Trade-offs:
Scope and approval
Refs #15616. That issue is a triaged bug. The triage named "the reconnect backoff from #14897, especially how foreground and wakeup resets interact with it while a host stays unreachable" as the likely fix area. It also said that a maintainer will decide on the fix direction.
This PR proposes one direction: keep the immediate retry, and limit how often a wakeup can reset the ladder. Other options are to not reset on foreground at all, or to use a different window. Either would change only
resetRetryLadderAfterWakeupand the test expectations.Verification
New tests are in the existing
packages/client-runtime/src/connection/supervisor.test.ts. They useit.effectwithTestClock, and every attempt fails transiently ("Could not resolve hostname.").retries at once on <reason> but keeps climbing the ladder, run once for each ofapplication-active,application-active-probe, andapplication-active-reconnect. Four wakeups during backoff each start an attempt with no clock movement, and the backoff delays are[2000, 2000, 4000, 8000, 16000]. AfterTestClock.adjust("5 minutes"), the next wakeup resets the ladder again: attempt 1, 2000 ms.restarts the retry ladder on every network change. After twoapplication-activewakeups (attempt numbers 1, 1, 2), anetwork-changedwakeup brings the ladder back to attempt 1.main(test only, before the fix): 4 failed. The app-active tests got delays[2000, 2000, 2000, 2000, 2000]and the network-change test got[1, 1, 1, 1], because every wakeup reset the ladder. The app-active tests had a longer title during that run; the assertions were the same.vp run check:changed upstream/main: format, lint, knip, typecheck (7 packages) and test (1 file, 34 tests) all pass.Not checked:
application-activewakeups is an inference: they match the reset mechanism, and supervisor activity shows up between attempts.Implemented with Claude Opus 5.5 via Claude Code (running in T3 Code).
🤖 Generated with Claude Code